Comments on Methods for Setting Confidence Limits

I discuss a number of issues which arise when computing confidence limits by frequentist or Bayesian methods. I begin with a reminder why P(hypothesis data) cannot be determined if the only input is the ‘objective’ data. I then discuss confidence intervals, with emphasis on the ‘unified approach’ based on likelihood-ratio ordering, and related methods. A number of issues arise, including conditioning, nuisance parameters, and robustness. For Bayesian methods, important issues are the prior and goodness-of-fit. I conclude with a list of items on which I think physicists from many points of view can agree. 1. PROLOGUE For most of this talk1, I assume familiarity with the ‘required reading’ for this workshop. But first, let’s review the root of the problem as I often explain it to students. (Imagine an oral exam.) Suppose you have a particle ID detector. You take it to a test beam and measure: P(counter says particle is ) = 90% P(counter says not particle is ) = 10% P(counter says particle is not ) = 1% P(counter says not particle is not ) = 99% Then you put the detector in your experiment. You select tracks which the detector says are pions. Question: What fraction of these tracks are pions? Answer: Cannot be determined from the given information! The missing information is the pion fraction in the particles incident on the detector: the initial P( ). Bayes’s theorem then tells us that P(particle is counter says ) P( ) P(counter says particle is ). All this makes total sense with the frequentist definition of P. Now suppose you look for a Higgs boson (H) at LEP and you do all the work to know: P(H signature there is H) = 90% P(no H signature there is H) = 10% P(H signature there is no H) = 1% P(no H signature there is no H) = 99% There is no problem defining these P’s with the frequentist definition of P. Then you do the experiment, and you have a Higgs signature. Question: What is the probability that you found the Higgs? Answer: Cannot be determined from the given information! The missing information is the analog of P( ): the ‘prior’ probability that there is a Higgs: P(H). Again Bayes’s theorem then tells us that P(particle is H H signature) P(H) P(H signature particle is H). * E-mail address: [email protected] I attempt to preserve the conversational nature of the talk in this writeup.